/subsystem=teiid:read-resource-description
The BufferManager is responsible for tracking both memory and disk usage by Teiid. Configuring the BufferManager properly is one of the most important parts of ensuring high performance. Execute following command on CLI to find all possible settings on BufferManager
/subsystem=teiid:read-resource-description
All the properties that start with "buffer-service" configure BufferManager. Some of them are described below. The Teiid engine uses batching to reduce the number of memory rows processed at a given time. The batch sizes may be adjusted to larger values if few clients will be accessing the Teiid server simultaneously.
max-reserve-kb - setting determines the total size in kilobytes of batches that can be held by the BufferManager in memory. This number does not account for persistent batches held by soft (such as index pages) or weak references. The default value of -1 will auto-calculate a typical max based upon the max heap available to the VM. The auto-calculated value assumes a 64bit architecture and will limit buffer usage to 50% of the first gigabyte of memory beyond the first 300 megabytes (which are assumed for use by the AS and other Teiid purposes) and 75% of the memory beyond that.
With default settings and an 8GB VM size, then max-reserve-kb will at a max use, (((1024-300) * 0.5) + (7 * 1024 * 0.75)) = 5738 MB or 5875712 KB
The BufferManager automatically triggers the use of a canonical value cache if enabled when more than 25% of the reserve is in use. This can dramatically cut the memory usage in situations where similar value sets are being read through Teiid, but does introduce a lookup cost. If you are processing small or highly similar datasets through Teiid, and wish to conserve memory, you should consider enabling value caching.
Memory consumption can be significantly more or less than the nominal target depending upon actual column values and whether value caching is enabled. Large non built-in type objects can exceed their default size estimate. If an out of memory errors occur, then set a lower the max-reserve-kb value. Also note that source lob values are held by memory references that are not cleared when a batch is persisted. With heavy lob usage you should ensure that buffers of other memory associated with lob references are appropriately sized.
max-processing-kb - setting determines the total size in kilobytes of batches that can be used by one active plan may be in addition to the memory held based on max-reserve-kb. Typical minimum memory required by Teiid when all the active plans are active is #active-plans*max-processing-kb. The default value of -1 will auto-calculate a typical max based upon the max heap available to the VM and max active plans. The auto-calculated value assumes a 64bit architecture and will limit processing batch usage to 10% of memory beyond the first 300 megabytes (which are assumed for use by the AS and other Teiid purposes).
In systems where large intermediate results are normal (scrolling cursors or sorting over millions of rows) you can consider increasing the max-processing-kb and decreasing the max-reserve-kb so that each request has access to an effectively smaller buffer space.
With default setting and 20 active-plans, if you have 8GB VM size, then max-processing-kb will at a max use will be (((1024-300) * 0.1) + (7 * 1024 * 0.1)) = 789.2 KB/20 = 39.46 KB. When all plans are not active then individual plans size can grow between 39 KB to 789 KB.
max-file-size - Each intermediate result buffer, temporary LOB, and temporary table is stored in its own set of buffer files, where an individual file is limited to max-file-size megabytes. Consider increasing the storage space available to all such files max-buffer-space if your installation makes use of internal materialization, makes heavy use of SQL/XML, or processes large row counts.
processor-batch-size - Usage of extremely large VM sizes and or datasets requires additional considerations. Teiid has a non-negligible amount of overhead per batch/table page on the order of 100-200 bytes. Depending on the data types involved each full batch/table page will represent a variable number of rows (a power of two multiple above or below the processor batch size). If you are dealing with datasets with billions of rows and you run into OutOfMemory issues, consider increasing the processor-batch-size to force the allocation of larger batches and table pages.
max-storage-object-size - This represents the individual batch page size. If the processor-batch-size is increased and/or you are dealing with extremely wide result sets (several hundred columns), then the default setting of 8MB for the max-storage-object-size may be too low. The inline-lobs also account in this size if batch contains them. The sizing for max-storage-object-size is in terms of serialized size, which will be much closer to the raw data size than the Java memory footprint estimation used for max-reserved-kb. max-storage-object-size should not be set too large relative to memory-buffer-space since it will reduce the performance of the memory buffer. The memory buffer supports only 1 concurrent writer for each max-storage-object-size of the memory-buffer-space.
Teiid temporary tables (also used for internal materialization) can only support 2^31-1 rows per table.
memory-buffer-space - setting controls the amount of on or off heap memory allocated as byte buffers for use by the Teiid buffer manager. This setting defaults to -1, which automatically determines a setting based upon whether it is on or off heap and the value for max-reserve-kb.
memory-buffer-off-heap - Take advantage of the BufferManager memory buffer to access system memory without allocating it to the heap. Setting memory-buffer-off-heap to "true" will allocate the Teiid memory buffer off heap. Depending on whether your installation is dedicated to Teiid and the amount of system memory available, this may be preferable to on-heap allocation. The primary benefit is additional memory usage for Teiid without additional garbage collection tuning. This becomes especially important in situations where more than 32GB of memory is desired for the VM. Note that when using off-heap allocation, the memory must still be available to the java process and that setting the value of memory-buffer-space too high may cause the VM to swap rather than reside in memory. With large off-heap buffer sizes (greater than several gigabytes) you may also need to adjust VM settings.
For Sun VMs the relevant VM settings are MaxDirectMemorySize and UseLargePages. For example adding:
-XX:MaxDirectMemorySize=12g -XX:+UseLargePages
to the VM process arguments would allow for an effective allocation of approximately an 11GB Teiid memory buffer (the memory-buffer-space setting) accounting for any additional direct memory that may be needed by the AS or applications running in the AS.
max-buffer-space - For table page and result batches the buffer manager will we a limited number of files that are dedicated to a particular storage size. However, as mentioned in the installation, creation of Teiid lob values (for example through SQL/XML) will typically create one file per lob once the lob exceeds the allowable in memory size of 8KB. In heavy usage scenarios, consider pointing the buffer directory on a partition that is routinely defragmented. By default Teiid will use up to 50GB of disk space. This is tracked in terms of the number of bytes written by Teiid. For large data sets, you may need to increase the max-buffer-space setting.